{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to compute graph metrics with Splink"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to the `compute_graph_metrics()` method"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To enable users to calculate a variety of graph metrics for their linked data, Splink provides the `compute_graph_metrics()` method.\n",
    "\n",
    "The method is called on the `linker` like so:\n",
    "\n",
    "```\n",
    "linker.clustering.compute_graph_metrics(df_predict, df_clustered, threshold_match_probability=0.95)\n",
    "```\n",
    "\n",
    "::: splink.internals.linker_components.clustering.LinkerClustering.compute_graph_metrics\n",
    "    handler: python\n",
    "    options:\n",
    "      show_root_heading: false\n",
    "      show_root_toc_entry: false\n",
    "      show_source: false\n",
    "      show_docstring_parameters: true\n",
    "      show_docstring_description: false\n",
    "      show_docstring_returns: false\n",
    "      show_docstring_examples: false\n",
    "      members_order: source\n",
    "      \n",
    "!!! warning\n",
    "\n",
    "    `threshold_match_probability` should be the same as the clustering threshold passed to `cluster_pairwise_predictions_at_threshold()`. If this information is available to Splink then it will be passed automatically, otherwise the user will have to provide it themselves and take care to ensure that threshold values align.\n",
    "\n",
    "The method generates tables containing graph metrics (for nodes, edges and clusters), and returns a data class of [Splink dataframes](../../../api_docs/splink_dataframe.md). The individual Splink dataframes containing node, edge and cluster metrics can be accessed as follows:\n",
    "\n",
    "```python\n",
    "graph_metrics = linker.clustering.compute_graph_metrics(\n",
    "    pairwise_predictions, clusters\n",
    ")\n",
    "\n",
    "df_edges = graph_metrics.edges.as_pandas_dataframe()\n",
    "df_nodes = graph_metrics.nodes.as_pandas_dataframe()\n",
    "df_clusters = graph_metrics.clusters.as_pandas_dataframe()\n",
    "\n",
    "```\n",
    "\n",
    "The metrics computed by `compute_graph_metrics()` include all those mentioned in the [Graph metrics](./graph_metrics.md) chapter, namely:\n",
    "\n",
    "* Node degree\n",
    "* Node centrality\n",
    "* 'Is bridge'\n",
    "* Cluster size\n",
    "* Cluster density\n",
    "* Cluster centrality\n",
    "\n",
    "All of these metrics are calculated by default. If you are unable to install the `igraph` package required for 'is bridge', this metric won't be calculated, however all other metrics will still be generated.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Full code example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code snippet computes graph metrics for a simple Splink dedupe model. A pandas dataframe of cluster metrics is displayed as the final output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": [
     "hide_output"
    ]
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You are using the default value for `max_pairs`, which may be too small and thus lead to inaccurate estimates for your model's u-parameters. Consider increasing to 1e8 or 1e9, which will result in more accurate estimates, but with a longer run time.\n",
      "----- Estimating u probabilities using random sampling -----\n",
      "\n",
      "Estimated u probabilities using random sampling\n",
      "\n",
      "Your model is not yet fully trained. Missing estimates for:\n",
      "    - first_name (no m values are trained).\n",
      "    - surname (no m values are trained).\n",
      "    - postcode_fake (no m values are trained).\n",
      "\n",
      "----- Starting EM training session -----\n",
      "\n",
      "Estimating the m probabilities of the model by blocking on:\n",
      "(l.\"first_name\" = r.\"first_name\") AND (l.\"surname\" = r.\"surname\")\n",
      "\n",
      "Parameter estimates will be made for the following comparison(s):\n",
      "    - postcode_fake\n",
      "\n",
      "Parameter estimates cannot be made for the following comparison(s) since they are used in the blocking rules: \n",
      "    - first_name\n",
      "    - surname\n",
      "\n",
      "Iteration 1: Largest change in params was -0.351 in the m_probability of postcode_fake, level `Exact match on postcode_fake`\n",
      "Iteration 2: Largest change in params was 0.109 in the m_probability of postcode_fake, level `All other comparisons`\n",
      "Iteration 3: Largest change in params was 0.0175 in the m_probability of postcode_fake, level `All other comparisons`\n",
      "Iteration 4: Largest change in params was 0.00234 in the m_probability of postcode_fake, level `All other comparisons`\n",
      "Iteration 5: Largest change in params was 0.000305 in the m_probability of postcode_fake, level `All other comparisons`\n",
      "Iteration 6: Largest change in params was 3.97e-05 in the m_probability of postcode_fake, level `All other comparisons`\n",
      "\n",
      "EM converged after 6 iterations\n",
      "\n",
      "Your model is not yet fully trained. Missing estimates for:\n",
      "    - first_name (no m values are trained).\n",
      "    - surname (no m values are trained).\n",
      "\n",
      "----- Starting EM training session -----\n",
      "\n",
      "Estimating the m probabilities of the model by blocking on:\n",
      "(l.\"dob\" = r.\"dob\") AND (SUBSTR(l.postcode_fake, 1, 3) = SUBSTR(r.postcode_fake, 1, 3))\n",
      "\n",
      "Parameter estimates will be made for the following comparison(s):\n",
      "    - first_name\n",
      "    - surname\n",
      "\n",
      "Parameter estimates cannot be made for the following comparison(s) since they are used in the blocking rules: \n",
      "    - postcode_fake\n",
      "\n",
      "Iteration 1: Largest change in params was 0.535 in probability_two_random_records_match\n",
      "Iteration 2: Largest change in params was 0.0869 in probability_two_random_records_match\n",
      "Iteration 3: Largest change in params was 0.021 in probability_two_random_records_match\n",
      "Iteration 4: Largest change in params was 0.00702 in probability_two_random_records_match\n",
      "Iteration 5: Largest change in params was 0.00306 in probability_two_random_records_match\n",
      "Iteration 6: Largest change in params was 0.0015 in probability_two_random_records_match\n",
      "Iteration 7: Largest change in params was 0.000764 in probability_two_random_records_match\n",
      "Iteration 8: Largest change in params was 0.000396 in probability_two_random_records_match\n",
      "Iteration 9: Largest change in params was 0.000207 in probability_two_random_records_match\n",
      "Iteration 10: Largest change in params was 0.000109 in probability_two_random_records_match\n",
      "Iteration 11: Largest change in params was 5.7e-05 in probability_two_random_records_match\n",
      "\n",
      "EM converged after 11 iterations\n",
      "\n",
      "Your model is fully trained. All comparisons have at least one estimate for their m and u values\n",
      "Completed iteration 1, root rows count 307\n",
      "Completed iteration 2, root rows count 49\n",
      "Completed iteration 3, root rows count 11\n",
      "Completed iteration 4, root rows count 0\n"
     ]
    }
   ],
   "source": [
    "import splink.comparison_library as cl\n",
    "from splink import DuckDBAPI, Linker, SettingsCreator, block_on, splink_datasets\n",
    "\n",
    "df = splink_datasets.historical_50k\n",
    "\n",
    "settings = SettingsCreator(\n",
    "    link_type=\"dedupe_only\",\n",
    "    comparisons=[\n",
    "        cl.ExactMatch(\n",
    "            \"first_name\",\n",
    "        ).configure(term_frequency_adjustments=True),\n",
    "        cl.JaroWinklerAtThresholds(\"surname\", score_threshold_or_thresholds=[0.9, 0.8]),\n",
    "        cl.LevenshteinAtThresholds(\n",
    "            \"postcode_fake\", distance_threshold_or_thresholds=[1, 2]\n",
    "        ),\n",
    "    ],\n",
    "    blocking_rules_to_generate_predictions=[\n",
    "        block_on(\"postcode_fake\", \"first_name\"),\n",
    "        block_on(\"first_name\", \"surname\"),\n",
    "        block_on(\"dob\", \"substr(postcode_fake,1,2)\"),\n",
    "        block_on(\"postcode_fake\", \"substr(dob,1,3)\"),\n",
    "        block_on(\"postcode_fake\", \"substr(dob,4,5)\"),\n",
    "    ],\n",
    "    retain_intermediate_calculation_columns=True,\n",
    ")\n",
    "\n",
    "db_api = DuckDBAPI()\n",
    "linker = Linker(df, settings, db_api)\n",
    "\n",
    "linker.training.estimate_u_using_random_sampling(max_pairs=1e6)\n",
    "\n",
    "linker.training.estimate_parameters_using_expectation_maximisation(\n",
    "    block_on(\"first_name\", \"surname\")\n",
    ")\n",
    "\n",
    "linker.training.estimate_parameters_using_expectation_maximisation(\n",
    "    block_on(\"dob\", \"substr(postcode_fake, 1,3)\")\n",
    ")\n",
    "\n",
    "pairwise_predictions = linker.inference.predict()\n",
    "clusters = linker.clustering.cluster_pairwise_predictions_at_threshold(\n",
    "    pairwise_predictions, 0.95\n",
    ")\n",
    "\n",
    "graph_metrics = linker.clustering.compute_graph_metrics(pairwise_predictions, clusters)\n",
    "\n",
    "df_clusters = graph_metrics.clusters.as_pandas_dataframe()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cluster_id</th>\n",
       "      <th>n_nodes</th>\n",
       "      <th>n_edges</th>\n",
       "      <th>density</th>\n",
       "      <th>cluster_centralisation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Q5076213-1</td>\n",
       "      <td>10</td>\n",
       "      <td>31.0</td>\n",
       "      <td>0.688889</td>\n",
       "      <td>0.250000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Q760788-1</td>\n",
       "      <td>9</td>\n",
       "      <td>30.0</td>\n",
       "      <td>0.833333</td>\n",
       "      <td>0.214286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Q88466525-10</td>\n",
       "      <td>3</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Q88466525-1</td>\n",
       "      <td>10</td>\n",
       "      <td>37.0</td>\n",
       "      <td>0.822222</td>\n",
       "      <td>0.222222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Q1386511-1</td>\n",
       "      <td>13</td>\n",
       "      <td>47.0</td>\n",
       "      <td>0.602564</td>\n",
       "      <td>0.272727</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21346</th>\n",
       "      <td>Q1562561-16</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21347</th>\n",
       "      <td>Q15999964-5</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21348</th>\n",
       "      <td>Q5363139-12</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21349</th>\n",
       "      <td>Q4722328-5</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21350</th>\n",
       "      <td>Q7528564-13</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>21351 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         cluster_id  n_nodes  n_edges   density  cluster_centralisation\n",
       "0        Q5076213-1       10     31.0  0.688889                0.250000\n",
       "1         Q760788-1        9     30.0  0.833333                0.214286\n",
       "2      Q88466525-10        3      3.0  1.000000                0.000000\n",
       "3       Q88466525-1       10     37.0  0.822222                0.222222\n",
       "4        Q1386511-1       13     47.0  0.602564                0.272727\n",
       "...             ...      ...      ...       ...                     ...\n",
       "21346   Q1562561-16        1      0.0       NaN                     NaN\n",
       "21347   Q15999964-5        1      0.0       NaN                     NaN\n",
       "21348   Q5363139-12        1      0.0       NaN                     NaN\n",
       "21349    Q4722328-5        1      0.0       NaN                     NaN\n",
       "21350   Q7528564-13        1      0.0       NaN                     NaN\n",
       "\n",
       "[21351 rows x 5 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_clusters"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "splink-bxsLLt4m",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
